The perils of using simulations to evaluate Massively Multiplayer Online Game performance

To test and benchmark Massively Multiplayer Online Games (MMOGs) requires hundreds, if not thousands of human players. Given this impracticality, many researchers substitute experimentation with simulation. However, when investigating performance and scalability issues, simulated experiments often yield results that heavily depend on the experimental setup. This paper critically reflects on the use of simulation to conduct experiments in MMOGs. Using Mammoth, a MMOGs research framework, performance measurements such as CPU usage, memory usage and used network bandwidth are collected while running the same game scenario using five different simulation setups. The results are analyzed, and the discovered differences are discussed. The paper concludes that experiments which are aimed at measuring the performance of a MMOGs using simulations must be designed very carefully, especially if they are run on a single machine.